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ABSTRACT 

The demand for computers to help manage learning 
comes mostly from those engaged in attempts to individualize 
education, who have discovered a need for computers to help theJi make 
decisions and organize their data* This paper considers the problems 
that can arise in implementing computer-managed learning by 
ind'^'^ating what has happened in existing, operational projects, 
Fi'^t, it describes some of the problems in diagnostic assessment of 
learners — the process of discovering what repertoire of behavior each 
learner has already acquired and where the gaps are in that 
repertoire* Second, the paper examines problems in the generation, 
analysis, and banking of test items. Next, problems in the selection 
of a particular sequence of objectives for an individual student by 
the computer are discussed, and finally, some of the problems that 
teachers and students experience in using the computer system are 
identified* (Author/SH) 
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PKOtiLti^S IN JMPLEiHEWTING COHPUTER-KAN/VGED. U.ARN ING eDucATroNiwEci^*fl% ' 

" i N*f IONAL iNSf irWTEOF 

,J ~ . ' EDUCATION 

^ , Introduction - ^ _ ri:^^-?^:.:^'?^.;^;;^ 

oo ^ ^ 

This paper Is about problems in using the computer to help manage the 

_^ ' * ^ 

CD learning process\^ It is ,jot jbout computer-assisted instruction (as that 

UJ ^ , ■ . 

term is §enera 1 1 y under s tood) , computer i 5ed class-schedul 1 ng , compute r i secJ * 

educa Li ona I accoun t i ng and f 1 nanc I a 1 con t ro t (Bushne 1 1 and A ) ten , 196 7) ^ or 

the 1 ink ; ng of computers in networks (Wcjngarten^ et^al** ^3ll) a 1 tfioutjh ^1 1; 

all of these the computer is indeed being used to help learning. 

7hi5 paper deals particularly with problems in four'areas: 

1) diagnostic assessment of learners 

^2) the genera t i on J ^^ana 1 ys is and banlcing of test items 

J) the ^-.equencirg o*"' iearnfng ruaterlals for indivrlduai lUudent.*? 

^'4) tiu- Inter-iace of students and teachers with th« computer 

Tlu^i;c -oroas were selected at the request of Richard Hooper^ Dlrectpr ol 

Nutiontil Development Prograntnie rn Computer Assisted Learning, for wliich 

the p/^|>er prtrpar^ed* 

X f'J: :.. ^ - ^'^ '--^ ^^^^ ^ 

Wh**rt* vif,>os tlie dofnand come from tbr computeti; c.o help maiioge. 1 t'^rn i nci? 

J[ certLihily does not arise in schoolii or collcjges in which, tradi t lonai 

^ M or.k ' t c^p ' nii*thofii; of" teachijig nrt^ employed. /Teochers who <iiisumo thac. 

. I ho \ V tijctt^n t5 rt^ c 1 050 i y s 1 mi I ar ^ nnti that tht*se iitudents will proceed 

.1; I he SL]nu* rate.by tlie sm\t\ r^outt; through tlie matter t<>' be learned, will 

<i I w rissuMR! thn t tlifc ccniputer huii I 1 1 1 le to of f er • 

"M)** drmnnd coH)es from those engaged In attempts to i nd I dua 1 1 se - 

"^AboLit ;;o year?; ago, Washburne el i\\ . {192:6) and Partchurst ^ 
rt*pc>rt^(t ihflt teachers using the' Wi nn^stka Techntqtie and the Dalton Plan, 
"iwo bcluMSK^ti for Indl vidtJt^Msed learning, found diagnostic assessment and; 
rf-co r d ^ket-p #ng f tmc t i ons n>05t t i*r»e^consijm! Hg ^ both for teachers and 
ptiplh,. ^Yiu tfie data volume In those schemes was small by comparison 
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with the volume used In two largos American computer-managed individualised 
educetron projects now in operation* Hor was there any attempt in those 
schemes to evolve sets of Instructional dec i s ion^rul es (to guide students) 
that used the data* 

I n the Un 1 ted States , new i mpetus was prov I ded for the t nd i v 1 dua 11 sed 
ticlucat ion n^ovemen t by Project TALENT (Flanagaric et aKtrl962)* . Project 
TALENT , itself made poss 1 ble by the computer ^ 1 n vol ved the "col )eci Ion In 
1%0 or ne'irly 2^000 Items of information for each of HO^OOO high school 
<;tL[donti3, Thy data were btinked in o computerj which subsequently produced 
a orr.fit number of ana!yses* The students were followed up one^ five and 
t^jn yttaf'j. hut ii was the initial analyses that deriionstratetl in tome 

detail tijc need for individualising education* Flanagan (1971) slated thtii 
M)t*iwt:en 2^% *ind 30^^ of ninth grade students had already ochievcd js ^much^ 
kriowlcdgr iind \jb i 1 I ty in such fields as English anrf Social Studies as t\ui 
f]V<:vng(T two I fth y radv p tudent . Th I s great var I abll i ty In the I t!ve To f 
artii t**/onirfi I of th*^ siud-iints in a particular clflss suggested vtrry rcft| ' 
fiend For i nd 1 v I dua I i sat i on - * . ' * P lanagnn o 1 so found tliot TALFNT dm a 
reflect<^d students* feelings .that their high school courses often did not 
\\\txrA their needs, ond thnt there was a \^t:k of stability ond lealUm In 
the $ t udent*; * occupat I onal choi ccs at thai t i mo . 

TALRf^T led to ih*i foundation ^f Project PLAN {Program for LenrnIn<J In 
Accordance with Needit) In 1966 by Flanasan* At about tim same t Uk^ dUiSe^r 
and others (Cootey and Glaser, l^Sfl; GIfliier and Nltcb^ 1970) In Pittsburgh^ 
conctived IPI ( Uid 1 yl dual ly Prescr) b^^d Instruction). These two art? major 
cornptii cr-^mnnayed I ndl v Ulunl I sed oducflt I on p rojec ts t nto wh I ch many m) 1 1 ion!i 
of dollar*; of development mon€*v have bCQn.poured. PLAN represents the 
most compi t:heniilve use of the computer^ offering complete curricula In ^ 
iM/iiheiuui , languiige, aoclal sciences and mathematics for both prlm^iry and 

st*t:ord^iry hrveffi. IP! Is a more llmlttsd schema* covering only msth- 
^matfc^, I/hvjtiaijft 3r..i problem-solving r^f^chnlqueii at the elemtintary levf^l. 
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The Open University represents a. 'utility' mode! In Its use of the computer* 
and does not offer as much I nd I v! dua 1 I sa t Ion as the other two. From the 
Lhree, a fine range of examples can be^provided of the problems that arise 
i n i mp 1 ement i ng computer -managed lea rni ng . ^'^ 
Four problem areas 

Computcr-mafiageci learning systems are built on a scientific* 
* btrhriv I aura ( 1 y-based mode). (This could be an adverse coiTuiientl) The 
:iysie^is are hjili by people who beJJeve-tt Is possible and useful to 
on.ijyfje, C':ut;gor i eie , measure dnd change' the behaviour of students (sec 
t:c)oley and G Lir.er / 1968 ; Anastasio and Morgan^ 1972; and Oaserp I965). 
Mo?;t o[ the problems in their systems arise in the! r attempts to carry out 
Uv.'stv fund i Oris - Frequently the protjlems are llie result of sheer Iiuinan 
dlvt^rslt.y: there are too many yarlabt<;s to take (ntd account- Even tht^ 
1 ,fU)0,OUit J)OC I tei-^is In the TALENT data bank cannot provide exactly the 
i n for^itii* t i cin rh;u i f. requ! 'cd for cer ta In I mportant dec( s fons ahout a 
.EnHvnt''; lo^ifnlng. Other prohlems arfs<^ from the need to ^)ur 
(1 i srr I ml naU oii'i » to ovtsr-cjene t^rt t i se from specific dato, and so on. 

Stich firoliiijmf; are not always solvable^ but this paper iu Iti'flSt 
i rui i cnt e , whai I h<*y are 1 1 ke In ^ikI s t i ng # operational projoc i s » First, 
U iJcscrili<^s <;ome of the probloms in dlftgnoatlc oss^.ssnienv of U^fi rnif^i^s , 
t1^/n IS, iho pr'o .ess of d I scover Inc) j^hat repertoire of liohaviour fiarh 
locirnc^r U<v.* *^1 ready acquired aivd where the gaps ore In that rttpGrtotrOp " 
a*^sunilng th^u lor him certain sets of educatlona:l obJeiCtlves ^nre deHlrntile. 
Sornnd, tiUs paiuM" exAmlnes prohlflins in the generation, 'Vnalysi^ 6»^d 

'[*j*utuiu^: Ihcrc rtrcf tievoral uch<sr projects that would yield further 
l'Al^mj^l<*^ of prfjbleins: ^^g.i the Computtfii^ ManaiJed loJirnlng ProJ<^ici at 
(:olurn(iLP Mat'<j University^ the Instructional Manflgonjant Sybtems CoursiS 
:u Wrv:i<-n^ U^^shhujton State College (uatt^ and Stratighun* lf)72) , und 
. (p<):-rJhly lio lono^^r operw 1 1 on«) ) Instructional MonageiDdni System 
(b-vf lrt[><-d !(dnrly by the Southwefit 'Regional Laboratory for Educational 
itr.t-jiith tuu\ D^vclopiiKira ain! Syst&rtis Davetopment Corporation* for I05 
Q Aiuit-lA lrttit*ntary ii<MK><>ls (Ct^ddes and Kool < t969) * 
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barikingof test items* Since test Iteins are intended to be valid for 
cer ta I n educat t on^l obj ect i ves , these obj ect r ves en ter the di s cuss r on 
again* Third, the ^electfon of a particular sequence of- objectives 
f'or an individual studerit by the computer contains a .-further set of 
prob 1 cms * f-'our th , thi L paper i dent I f i es some of \,the prob 1 ems that 
tci^chers and 5tud?^nts experience in using the. computer system; these 
£:re conmionly called Interface problems* ' ^ 

Problems of diagnostic assessment 

Fundamental to eny computer^menaged.. learn I ng systernthal attempts 
Lo iiiatch objectives and materials to learner characteristics and needs 
art: anaiyties of each of these: learner characteristics and needs, 
loari^uj objectives^ and learning materials* Diagnostic a?isessment 
f M vu i vo-; f dent i fy I ny line I earner* s character i s 1 1 cs and needs , \ is f \ r^t 
purpose i !j 10 niake possible logl<:al selection of npproprlotw obJoctJvt;^. 
r<nr tlu^ l<!cirnor, thus cjuldlng him through tj^u syVttejn tOi>fflrclft c^onts he nifiy 
{ :\k'(: rionit' sitiril I , pfirt in doterml nlng , Its sticnn^l 'piir[-iOi^<* i . mt\kt^ 
po';slhlc I'OcilccTt s<decclon of appropriate materlnla that will offer lUo 
i('f.^rnT?r a vcirtety of routes through the knov-Jledge at a p:jc<.i taryt.*ly 
dtrtcrminod by himself* Just as the good teacher wishes to know, hi b 
indiv?ttua) pupils well enough to take Into account their pecullnr luusds 
so in iho coniputoi'-managed system a data bank of student chftrzictort is t1 cs , 
Includlru) d<uolls of recent performance on tests. Is Intended to ^'ortn the 
b/js/s (or ' Indl V (dtml ]y prescr f bed ; rtst ruct Ion or' for Mearnfnp In 
accor danr<^ , wl tK needs ' * . / ... 

Tini first problems arise |n the computer-bflaed tiyntem when th*s dl*tfl 
Uiw^k hrtii lo^Uo filled- Consider the Open Un I vars I ty ; the slmp)e»st system 
oi' the three ex«-np let* when Its d«*ta bank of studant chftrrtCterl^tUs U 
concornod. The benk contains, for aach students a minimum oF biographical 
data cbilt^cted on his application form. Few» If ainy, of the d^t^ills itre 
f>r v^Hu<^(or tho*.e who construct the course*^ anU- none rtt all art* t^ikisn 
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into account to determine his course of study. Once he starts studying, 

In b test results are also banked; some of this set of data, does at least 

g^j to t^^^ LdLpr and counseUorj, and may help those people to help him. 

In kLAN and \P\t students provide large quantities of data through 

c/ar:nsiv(' (.ebting carried out at various times but particularly at the 

i Hij LiU' si:hool year. For exemplei in PLAN, a student cakes an annu^J. 

Lrvt [jottLry consisting of l) an iS^^scale Developed Abiiitit^s Perlorniancc 

1 L^s ; , ;?b*Hu^.s t n^ 5»u ;h var i ables as vocabul ary development , readi ny com- 

[: !iri>-i;oi^ .^H'j <3h5^tract reasoning ability^ 2) a 3^"scale General I n forn^itt i oh 

i e . t la dr ic rtni ne h f s pat tern of f unct i on a I 1 n teres ts ; 'i) a I 2-s ca I e PLAN 

hMrf'i:'>t Ijjwvuor'y to ciscertain areas of potential or exprcasc^d inicre^it; 

.1 StuJt^fU Altitude Inventory; and 5) 3 battery PLAfJ AchiL^venicnt 

l.^E'.. lh:3 [)iuterv Is fn addition to the tests hf* Uiia-^P (of t'cuJi tfie 

iv;('r,\y idodulos he studifts each yonr in tit letist four ^i\\jt>r sctioo) 

."^jt^f.M'. tn PLAN, tiata are also collccied Vrom thr tn.K:ju-i ot>oui iht 
i 

■Jr i(' .^l^ilily I.C utM'k f ndt^pt nd(rn t ^ V . 

f ht.*i',' /TTc t hrer k f nds of prob I cm rhdt tir I $c , Thoy 1 .n c tn t \\c 
yv^i\f tit r.nMrrtlfMi, th«^ apprOf^r 1 c^tcnoss of tho (<?^ti,, anc/ th<' u.^y^ 

in vvliiUi tilt; (iala arc U5ed, * ^ 

J f rM 1 y , tilt' mode* of data col 1 «c l fon is optMi 1 1% r r I i 1 r I mh. . I':m t tM" 1 
f^.iprr- -^iiitl-pt^nct I lasts a?> usecj !n PLAN seem the only pr/ictica! Wjiy xo 
I n \ \o( \ t ar xUe computer- bank ; 1 on g thy rue tiiral 1 nt erv I ows , 

v.ii hifi tJa^srnoni beiiflvi'our and other psycho-boci oln<i f r^*! moans cvt 
tnrnstnlnf] ;n't* fj<inoralty out of tht^ question- V<*t'th** b^tt i t^r' f t^^i Induce^ 
V 1 1 i^fii} 1 1*; I -t>r I un( at I on In s tudent 5 (even adu 1 1 Op«n I v<*i n I ty s tude*nt s ) 
•Ai Huh wumy fc*jaid th<' t**s»tti f\s the most Importitnt part of the proiiramme^ 
i^iiJ cHily »rt|arcl the learning materials as worthwhile If tht*y r<*lAte cln:^e!ly 
\ii thr t(*M^. This Is undealrsbla^ since even large l>ttttftrleiv supply cinly 
MnUif-it ^^nnpllngs t>f btudf^nt ACtttud&s ftnil performance* 



Secondly^ the tests are of ten ^ I nappropri ate , so that tHe data are of 
liLtle use /or diagnosis, Many of the test results lesve t he i ns true tlona I 
designer saying *so what? * ."l'' For example, what use can be made of an 
abstract retisoning score? if the score is low^ can the studLsnt bti askc^cl 
Lo complete I earn ing modul es that will take the score Into account^ or* 

^'produce an increase in it? The answer ?5 no» beca^usc no modules fiave hitVA\ 
tlcjjiyried with that particular construct" in mind, nor do ) ns l ruct i oiu"il 
df/Tiignc^^ have, much Idea of how to put together such TOdiiles. THino ivt- 
n rL*w<ireaii, sucli as ruatheruat i cs , where the niBtchlng qT test ^scores ond 

^learning materials Is a little more straightforward, biU twen U\ thnso 
tiic^i^i rhe dtscl s I on- ru 1 es have not yet been devised for th<! loiiriier to 

'iijpposedly diagnostic^ scores to appropriate Icnrn^tuj nvi; tnr I n 1 ? . 
Cr>oley TJflser (1969) admit these dl ff Icu! t ifts in the IPI projf^LU. 

fiiirOly, tfic data easily gec'ml sused, Becouf^e tht^ tests y^eld 
i|tujnt I { ( t^d d^atti, tht^se data quickly assy me a validity and rcl.ltibMIt/ fni- 
l.t'/nivl ( tifj Ifrv^Hs thai" would bo quoted by the tost HeMgni^r-.. ^^Inro thr 
fiiiii'.) linVc ro !'t^ u^yjd to guide difficult dectsionj about modules Ui lif. 
>Lucifinf, thr temptation to n>ove Into ^ (n<-cfian I s t i c inc :c c^f thoutjlit hi vtf'v 
'.iruiKj. ^ [he. clato :;oy you are a 2h7h type, theroforo you svhoultl &tudy 
modulo ;f?[l^- 32, * This tendency Is t^elnforced in PLAN, wlierc rht* 
c.f>mpttt<*r Is supplied with ei set of decl s Ion- ru )es rel at I no^hlefl y to whcit 
lUr ^ifufltrnr already know*>. p^lus hi? pflco of stuct)^ rhe' rulos la^d-to tho 
]nndu(:iiL>n l»y the computer of pro^ranifiHj of Study foi* the >(ude?^nt for the 
(riisutnt] (jUttf^er or year. It Is true that )n PLAN (j^ind at rhe Open 
UnlvinsMy) the computer can be ovor-ruled by the t^i^ich^i^ tut thts computci 
nppotj rs lo hftve far niore diagnostic data fit lt$ fingertips. Iti 
tiiiQgrM 1 ni>6 ara usual 'y followisd, backed as they ere by about 'tU setionds of 
cnmpuirr ftfprestsn t ( HQ ^ntremaly large number of deQ(?*lons for each 

lii'^drni. ih**i r u I US thomtiti 1 viffl , however, are rather prlp*U1vai If a 
M udfsiu hcitj complesttsd moduUft X and Yi h<*. should nriw do In so nmny d^^ys ; 



f f hi'j re.adlng level is tielt^w a certain score, he should use the^easy' 
jn t t (iT one was prepared for that module); if he has a record of poor 
mdcpendt^nl study habits, the teacher should be remmded q\' tliis for 
units rerfuirint] niostly independent study, and %o on.. 

A more rtexible but also aiore limited system of conlput^i2r*^)t1iit:d 
dirtcjnoGiic assessment has been developed In a few smalt projcats uj:lng 
ofv Mnc icsting, f-'or example^ the Un i vers'l ty of Calffornla til 0£) v i 
Ih'fs tjivon Its studc!nts In-anima! and human physiology access to 
co'npLih^r termfnal on which they can trtst the^ r knowledge* of' ronr^w 
fiiaterial ond receive advice un remedifsl wcJrk, (Walters ^ R.F. <il., 197^:). 
Ihis pr'o]f*ct i he^ivi ly dependent on tutor support on a ratio o\' \:} up 
lo l:i:^, thortiforc It cannot easily be. romptired with PLAN ^ 1^1 or i\M: Opt^i 
Uni ViTsT \y. \ 

Pvob U-ifis in tita gcnerotlon, ^rialy E-j s find banking of t ( ji; t - 1 1 e ins 

Stolurow (l*.)65) was one of the first to use ihc conipiiver to ^Mtivcnt*' 

or nnnf* rn I o test 1 terns - Hf ? I nt«-.rf*ji t . Uy in thr d f rt»ct f on n f us i imj r ■ 

^.piHtM" to ho If gencr^ate programmed learning sequ{*nc(VJ , partlculnriy f 

nui : ^'it^mn i i cs.. Sovpj'n I reports were pub li sh6d '^^ the 1 atL' 1 *1f)0s by 

Su> 1 urov^* ,in^l li f s:^co 1 I e agues at the Un i vers 1 ty of 1 1 I i no I s , hut i fK^r^-<; -\ i uJ \ t-- 

riiu:. t: fni r"(M,)arH<ji:i ii£ strictly e«perln>ent*»l work. 

Su^>|H^;r, nt Stnnford Un I vers I ty^.( see Supp^s ond Horn I n^|S t J^r , P.)/OK h^i^ 

cxptM i nienfofl wl I Computer genera 1 1 on of test - 1 xem for inflt htim*u I c s ■ 

hKjfc I}) hf^ *.(^rlo?> of appHtid CAI studies. Sttarle^ hts chlrf cduC^tlon/U 

d-*^lMtiMt,, lnthcnLed (In a personal Interview In August ^1973) th«t this h^d 

bcru onr ol I he l*i*uT. succeb&fut aspcccA of Ihel r atudlas. She emph^isiLs^^d 

(!wit tjrt t FuTTximputiir to the staye of gen<?rat 1 rig test-ltcn^s n fl^^al 

oi |.Mcl I nil i»n ry poddQoglcal work was required. ^ In th<? process this ^^^r^rk 

U Wit^i quite likely that th<; i nves 1 1 gator woy tU dUcover that th^> 

I ro I ru*. I ( or^n 1 tnatcirlals were not rich enough t*^yletd ci larcjt? niitntit^i" or 
) ' -■ 

[(^L^inr... L:6mpui,r;i; y<inc riit 1 <;>n i^t I C 6:11;? d<^pQndti OH th^ eHf5t<^nC^ Qt" ^^V^^^r^J 
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i'dcLors or ^'ar I ab I es that can be man I pu 1 a ted . For %om s ubjc^c : t\\<} c l^t , 
Se£sri,3 had found that only trivial variables were available for man i pul a l i on . 

In Searle's opinion, It was very expensive to generate Irenib by com^ 
pute r (ind se 1 dom wa s the expens e J us ti f i ed by the res u 1 ts • Sh^i thought 
there were relatively few situations in which such a } arqe variety of iiemr 
ui^s required. Even the Open Unl vers I ty ^ s deraand for a f7 umber (up to 

ttis ts cover I ng the same ma ter i a I f for lise In d I f ferent yetirs , was probiJ^j 1 y 
loG small to Justify computer generation, 

A further problem arose in connectron wl th the compuLsr-gcinerntcd * icww^ 
L n Suppes' project: in the schools^ It proved difficult to persuade pt'o;^]^^ 
wJio^juiu not set the tests to accept the results for students tht^y hr>d 
w^utjht. this was possibly a general ^problem of teacher attitudes towfifd^^. 
Ht: cxterMril agency^ but the teachers felt that the tests werr> o^ten 
ir.^;pl>roprUite. The I tems^ may indeed have been a poor sampling of the 
^.uiriculum tauyht by those teachers " cv6n If the Items were reprcse.n trtt I vr 
il i>omr ivfitten curriculum. If the Items were not reprr^fientat 1 ve » then ^-^^^ 

;.\.uid [^ave Just complaint that the tosts were f^hpnr/*ntly r.on-- 
Mollin^j the curriculum, As In many Instructional sltu^itlons^ the 
I r i.jLi onsh ? ps between curriculum^ objectives and test Iteriis may not hftv<^ 
i'*" n -Its good as they should hav« been. 

Wi)*Mhor or not the computer Is i*sed to generat^a test lioius^ I: <.vrr. 
^ vtvnJnly !iclp to analyst* thom^ both !n tertns of* content and perforn^^jncr 
(h^'iLirt , 15)71). Content analysis of ^lt«m» Is carrlad^put fot- Indexing 
fitrrpnr.<^?> , ond the computer Is usod simply to provlcte a f^^i /icc^s;; systf^rn. 
Huts within an I tern hank there will bfl groupings ofwl terns by subjoct-mottn , 

<ours<' l<^vul J questlon^typCj and so on* A comp)c*t0 test Ciin be put 

[ ■ ^ 

tofjct fuT otxording to any set of sped f'lc*t I one using thf>$^ <)roup.l tigs , 
onct! there (ire sufflclflpit items In th« bank. The problems of compMfltlon 
originate in dl f f I cul t l«i of content nicognttlon (see Garbner^ at al., 
1963) and selecting suitable dsflcrlpton, difficulties not pacullar to 
^^omput^rs (Commltteean Information in the B«havlorat Sciences, 1967). 



1 

it 

The content specifications would Ije regarded as useless^ huwever, 
without performance data-^ Hany years of psychometric work have to 
^ set of standard requi rennents of data for Items In a tes t" i tem bank. 
These requirements include difficulty end discriminate indices, 
reliability indices, and descriptions of the samples on which these 
i nd I ces are based. The^ d I scr i mi nat i on J ndox s hows the. exten t to whi ch 
studc^nts who do well on the test as a whole do well on individual items: 
i.e., does the Item discriminate between. good and poor performers? The 
reiiabMity lnde?t reflects the consistency with which the item appears to 
tes t pe r formances ^ for examp*. e , an ambi guous 1 y -worded i tem Is likely to 
yield loi^ reliability. The descriptions of samples used In deriving the 
Indices are vital for the test compiler to be able to Judge whether the , 
items art! appropriate for the group he Intends to have tested. ^ 

Ihore is no problem in providing these basic performance dato througt^ 
the computer, which Is Ideal for analysing the scores of students uho hove 
.ilroridy taknn the test Items- This is stenderd practice In some North 
f\\vrv]i:<-}t^ t}n)yf?irs\\ \vs , uh]cU nrmkowlde use of objective ?rst'^. In fi^cl^ 
-the fjn^7ipul.<M- can quite easily provide o number of i^dd! t i ona 1 hidicfts, tjsofuj 
to lUa f.est e?t;thor Fn. reviewing his items and In deciding whlcl^to oMnTlruitt*. 
or revis<.: (i'horndJke, 1970- . A quits sophisticated syiitt'm of thi? kind Is 
bi:hKi prepfirecf for use in the Open University (for an outline of trho pro- 

ihti <■^J>^<;^ce of ^ computerised tei;t-ltem bank I si that the lilstltutlon 
has nut only ji reserve Rtore of Items ready to b« Hssembled into tesr?i^ 
short notice If nftCtt^sijry, but i^lso fl store of Item-t of known ch^ ract g r ht I cs . 
Yot th«; lol lowing eKiimpIes Indicate that frequently this p.otnt U IgnortxI. 

Ai n Junior collogo InJolilIftt, llUnols, ct mini-bank of I tens- was cJaveloped 
on a small computer to serve the needs of a biology for multiple versions of 
a tout n->tsi In nn fludlo-tutorl a 1 couree. The report on thU work (Wagner 
flhd ni«c(t. I!)7,2) IndU-Ates that Items were selacted at random. from the? benkt 
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not on the bas i s of known cha rac ten s 1 1 cs • Th i s is not real 1 y sa t i s factory • 
A bimtlar bank at Washington State University lacked not only item 
characteristics but yiel ded ^ston i sh i ngty low re 1 i ab i I i ty coe f f i c i ents 
(Hammer and Henderson, 1972) because of poor psychometric design. The 
EXAHIN^^t^es t-f tem bank used at' the College of Business Courses at the 
Univcrr.iCy of South Florida is also a primitive tool, psychome t r f call y 
speaking (BIrkin, 157?)^ bank developed at Iowa State University for 

ci biology course did at least have difficulty estimates attached to each 
item, but theso wtM'c not refined on tho basis of student performance 
(Franke, et aK , l972) ^ 

The: value of knowing the characteristics of tlie i tc^ms ^ at least in tcrjns 
of given test s amp les, Is that tests for different purpO£;es or para I I el 
forms of 0 test for the same purpose^ can be conlpl led, For example* many 
unfvorf.!t.y t.e!;ts compiled Intuitively have ^ l^rge proportion of medium- 
d i f f i tu U y 1 Lems ^ yet ihe ai ms of the examl ner$ I nctude.^ I sol at i ng tho 
exccllfMn. nnd the falling student^:, The dffferenct^ in practice between 
rho fnilinfj aru.1 passing students mtiy lie in one vital llem^ ffiMod by Uir 
f?iilur<vi ^nd passed by the rest. In^;tead^ tt would be beHtM' to have I to-ms 
o\' t h*it I cvo I f)f d i f^r 1 cu I ty and fewer middle-difficulty It f^ms • 

In summary^ item banks have a disappointing record. The probU-.ms are 
not I (iftuperab le ^ but ' few i ns 1 1 tut I ons have t r I ed to tack I e' them. 
Problems In th<3 stftquenclng of learnlnjj mai^erialfi 

Urunrr (l!)6o) suggested^ that schoolchildren of any age could be taught 
any concept, admittedly at dl fferr.nt levels of understanding, i^egn^i (1^6^^, 
15)6fi) ► on the other hand^ argued 7cr the hierftrchlcal nature of learning 
tasks* In a rdc«!ni ssomlnar* Pask proposod that a nodal structure be 
IdentlHt^U for auy flrld of nubject-matter ^ and thfit learners may enter that 
siruciure^at many dlftere^nt joints with. fair chancen of success In l«arntng 
I Lb con Cfints , 

Q Of thttst^ three auchorltle&i Gagne 1st the one chiefly aspou^^d by chose 

AC 

MB working wlih computer-managed learning* ThU fs not Surprising, since 
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ls able to supply a niorc dogmatic set of decl s ! on-^ru I cs for bcqucnclng 
learners' use 'of materials. 

The basic problem in using the computer to determine the sequence 
each learner should follow 's one of finding reasons for reducing the 
Options open to the student at each decision point. For example^ if a 
student in PLAN has Just completed module 21-5^0-2, Mul tlpl I cation and 
Division of Fractions , what is there to prevent him from going next to 
any module In PLAN? For a series of ten objectives, there are over 
3 million possible sequences. 

To begin with, there are practical restrictions. For example, the 
supply of instructional materials In any one classroom is limited by cost 
and space consideration. Third graders cannot study high school modules, 
and even wealthy American schools are forced to assume for economic 
reasons, that no more than a dozen students will arrive at the same 
module simultaneously: the cost of equipping classrooms would otherwise 
be very high indeed. 

In addition, there ore theoretical restrictions?, chiefly In mathematics, 
when the sequence of learning tasks Is thought to be critical. Even then, 
however, it is In^terestlng to note 'that there was some disagreement among 
the mathematicians working on PLAN about the sequences to be followed. In 
tho event, the chosen sequences are similar to those In the available tt^xt" 
hooks, because the student Is less likely to become confused if he follows 
themi 

For riK>st of the modules, however, the sequencing rules are extrentely 
arbitrary, and my actually detract from the i ndi vidual ! sat Ion of learning^ 
since the student will hdVe tp Ignore the computer's authoritarian 
instructions In his programme of study If he wants to change the sequence 
written Into the computer for one that he likes bettor. 
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Dunn (1972) reported that in PLAN^ a student's programme of sttjdies 
would be determined by the computer taking into account his long range 
goal s (vocat iona J J ntert'j ts) , state and local cur r i cal um requl rements , and 
his past achievements. His paper certainly does not explain clearly how 
this is done. He escapes belnj, specific. * From all that has been 
published on the^ use of long ranga goals In Pi.ANi theriii is no Indication 
that these goals Influence the sequence of studies except possibly at 
secondary shool levels where some subject areas may be deliberately 
excluded or included. State and locdl requirements have a similar effect: 
they may require that U.S. hi story shal I be taught to all el Cjhth-graders ^ 
fo^ example. A student's past achievement scores are also unlikely to 
Influence the sequence of studies^ except by accelerating the student 
through subject-matter fn which he has excelled and decelerating him In 
areas In which he has failed beforii. 

I n summary ^ the computer does little to resol ye the perenn I al probl ems 
of sequencing faced by all curriculum reformer;* and designers. Decisions 
about whether a student learns set theory before he learns vulgar fractions 
remain? In the hands of the textbook writers even *n PLAN ^ since the 
nodules are basically sets of Instructions for using poitlons of existing 
texts. Certainly both PLAN and IPI are far from the stage of being able 
to use the computer to manage the sequence of learning for each student in 
Q fully adaptive fashion. 
Interface problems 

People tend to blame machines. In computer-managed Instruction, th(s 
computer comes In for more than Its share of blame from adml n I fttrators ^ 
teachers ond students. Many of the problems can bo tracc>ctwto the Interface! 
or the point at » ^^Ich the people actually have to u,:e the machines by 
communicating with them. Interface problems arise mainly through Incorrect 
signals being given to the computer. 

mc 
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These incorrect signals are of two kinds: incorrect programming of 
the computer and incorrect data. The programming of the computer *rs in 
the hands oF the developer ofthe system, whether It is the Open University 
or American institutes for ResearchVWesLi nghouse Learning Corporation 
(for Plan). Errors in the developnjeai stage may cause problems when the 
system becomes operational, but these can be remedied in subsequent years. 

The secG.id type of error, incorrect dota. Is much harder to dea) with, 
and deserves special attention here. The Issue th^t is at stake is the 
validity of the'^^^SlPcaVil'ase used to reach decisions, (mmense as it is in each 
of the three systems, PLAN, 1PI and the Open University, The problems to 
be solved are those of rendering the base more valid. 

The first point at which learners meet the machine, as it were, is in 
entering ihe system. The applicant to the Open University receives a form 
designed to be data-processed. The limitations of the computer require him 
to code for the computer data about himself, about his occupation, for example. 
Host applicants are not accustomed to doing this. In PLAW, students taking 
the initial tests not only have lo use an (ubiquitous) identification number; 
they also have to use \8H score^sheets that can be read by a mark-sensing 
terminal. These sheets not only require unaccustomed responses but also 
limit the types of questions that can be asked quite severely (27 five-choice 
i terns ) . 

At the Open University, in PLAN and in *P1 there are checking routines 
built Into the prograrr^.Tit ng to ensure that certain types of errors have not 
been made. Thus the document-readers will reject an incorrect Identification 
numberj^ will reject a score sheet on which guessing student has mferked more 
than a cer ta I n number of * s 1 ots * for answers , end will reject a test for a 
module or course the student is not registered as stLfdying. 

None of the error-detection systems Is entirely foolproof, but the 
estimates by the Open University computer administrators are that less ' 
than 5% of the documents enter the s>titemwith errors. This estimate rnay 
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be on the low side, however, since other documents may be filled in correctly 
from the computer's point of view but represent a range of errors on the part 
of the student in completing the document. For example^ he may have skipped 
a line or two by mistake, thus throwing out the whole pattern of marks on the 
rest of the score-sheet. Computers can be programmed to search for this 
type of error, but at greater expense,. 

!n PLAN, most schools have their own terminal, an i BH 2956 optical card 
reader plus an 1 6H 27^*0^ cont rol un' t and teletype/ Some of 
those involved In developing PLAN envisaged that students might have direct 
accesf> to the terminal, but it soon became apparent that nornallyonly 



operated from a central computer {ad one time. East and West Coast terminals 
were linked to an )6H 360/50 In Iowa), the cards ar^; read, and the otitput 
for the teacher anJ his studr^nts is printed on the teletype, ready for the 
next day of school. The readers can only read 12 words a minute. These 
are too slow for a system that will have 60,000 students In 1973-7^, but 
faster terminals are nore expensive. Slow terminals mean not only high 
line costs but also delayed turnaround, which in turn may lead to reduced 
use of the computer facility. Indeed, Westinghouse is now selling PIAN . 
as an i nd i vl dual i sed sy s tem wl th or wi thout the compute r-mnagement , That 
decision was of course guided by commercial as welt as educational motives. 

In IPI, a POP-I5 computer is at the core of the system, being linked 
to IBM 1050 terminals and I BH 1232 scanners in the schools (Glaser, 1969)- 

Klanagan (1970) certainly pinned his hopes on PLAH students being 
able to use the computer to help them. He wrote ^t is possible that the 
main social impact of Project PLAN will be through its computerised guidance 
system by which it is hoped that the students will formulate reasonable 
goals, develop plans for achieving those goals, and take responsibility for 
carrying out those plans^. Yet* as we have.3een> the articulation between 
riie battery of test;? used in PLAN and the I nstruct I oi al programme is weak* 



teachers or clerks would be using 




During the' night, the terminal is 



The computer absorbs a r.iass of daLa» and by some clevious magic» suggests 

a programme of studies. it is left to th* teacher or counsellor to 

explain, if he can^ the maqic. This>i5 one of the critical problems of 

interface. It can only be solved through es tab 1 i sfi i ng in the minds of 

tcdchersp counsellors and stu'Jents ihv validity and reliability of the 

processes within the computer. There are those who will believe the 

computer printout simply because ic Is from the computer^ bur there are 

ethers who wish to have the processes explafned. On explanation and 

analysis^ the processes reveal njany defects. The fact that human 

pfanners and designers were responsible for these defects^ tind that human 

teachers and counsellors have many defects too^ does not solve the 
> 

i nte r f ac^ probl em. 

In the Open UnLs^rsity, similar problems of interface persist^ 
chiefly in the oornpul 4M"-markeci ^^sslgriment system^ but also in the survey 
research c^irr^ed out ^>y the University using computor-read ques t i onria i res , 
Among students there is a basic fJisrrust of the computer * s 'abt 1 i Ly to 
score the tesis correctly; thib is in spi H3 of the fact that in every 
single coiSe t students' appeals ^'^ga^n^t feolty scoring by the computer uawc 
proved groundless. The students have not had great difficulty in learning 
how to use the forms^ however^ and accept the computer-marked assignments as 
part of the continuous assessment of their year's work. In the survey 
research^ there have been the usual problems of students completing 
ques t i onn^'^i res for television programmes in weeks when there were none ^ 
and of misunderstanding directions for filling in the answer sheets for 
the docuinent-reader. It is hard tc assess whether 'hese problems originate 
fTlt^i n I y in the f ornta t demanded by the reade r (and hence the compu te r ) , 
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Conc 1 us i on 

This paper has not listed every probU^ni that has occurred or been thouglH 
of in attempts to i r.p I emtin : compute r^managod learning. Nor has it provided 
a taAOnomy of such problems, insteed^ it hss discussed a selection of real 
problems that have to be recognised by those who wish to realise the 
computer's immense potential in helping to rudn^tge the learning process. 
The criticisms of the Open University, PLAN one iPl reflect upon areas whero 
problems have not yet been solved, and fgnore deliberatety the. gtei^i con* 
ti ibutions that each of the three have mode towards improving learning. 

Finally^ this paper is by no means it compreErens i ve review of the 

relevant literature. It is v^/ritten t rorp the viewpoint of an educational 

psyciio togi s t , not a computer specialist, and or.iits references to technical 

aspects of the c^umputfng or df*ia processing e<;uipnient, for examole. It 

also onii t s niuch re 1 e van t t heorc vical iitt;ratuf'c on cyfTicul um deG I gn , teb 1 1 ng 

and eval ua t. i on . Next t i n'c ^ piJ rhop v c^ Lihnrt book wi U be net^ded > rather than 
i 

a sfTOrt paper! 
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